HDF5

HDF5 stands for Hierarchial Data Format. This file structure is designed for scientific data. This is a basically a file system set up in a single file.

A good resource for HDF5 is The HDF Group documentation can be found here https://portal.hdfgroup.org/display/HDF5/HDF5

HDF5 files have 3 parts

1. Group - this would be the file folder it can be empty, contain one or more groups, contain contain one or more datasets, or a combination of groups and datasets with supporting meta data.
2. Dataset - this would be the file in our file system structure. IT is a multi-dimensional array of data elements with supporting meta data
3. Attribute List - This is an optional user defined HDF5 structure that will give more information about the object it relates to it can be added to a group or a dataset.

Datasets

Normal Datasets are created with a specified size with a specific number of X columns and Y rows or specified data. They can be created from a numpy array or a pandas dataframe. We can create an extendable HDF5 dataset Note this does require chunking

According to the HDF Group website https://portal.hdfgroup.org/display/HDF5/Extendible+Datasets The following operations are required in order to extend a dataset:

1. Declare the dataspace of the dataset to have unlimited dimensions for all dimensions that might eventually be extended.
2. Set dataset creation properties to enable chunking.
3. Create the dataset.
4. Extend the size of the dataset.